Systems and methods for video/multimedia rendering, composition, and user-interactivity

ABSTRACT

An interactive video/multimedia application (IVM application) may specify one or more media assets for playback. The IVM application may define the rendering, composition, and interactivity of one or more the assets, such as video. Video multimedia application data (IVMA data may) be used to define the behavior of the IVM application. The IVMA data may be embodied as a standalone file in a text or binary, compressed format. Alternatively, the IVMA data may be embedded within other media content. A video asset used in the IVM application may include embedded, content-aware metadata that is tightly coupled to the asset. The IVM application may reference the content-aware metadata embedded within the asset to define the rendering and composition of application display elements and user-interactivity features. The interactive video/multimedia application (defined by the video and multimedia application data) may be presented to a viewer in a player application.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/962,563, filed Dec. 8, 2015, titled “SYSTEMS AND METHODS FORVIDEO/MULTIMEDIA RENDERING, COMPOSITION, AND USER-INTERACTIVITY,” whichis a continuation of U.S. patent application Ser. No. 14/524,565, filedOct. 27, 2014, titled “SYSTEMS AND METHODS FOR VIDEO/MULTIMEDIARENDERING, COMPOSITION, AND USER-INTERACTIVITY,” which is acontinutation of U.S. patent application Ser. No. 13/972,013, filed Aug.21, 2013, titled “SYSTEMS AND METHODS FOR VIDEO/MULTIMEDIA RENDERING,COMPOSITION, AND USER-INTERACTIVITY,” now issued as U.S. Pat. No.8,948,250, which is a continuation of U.S. patent application Ser. No.13/742,523, filed Jan. 16, 2013, titled “SYSTEMS AND METHODS FORVIDEO/MULTIMEDIA RENDERING, COMPOSITION, AND USER-INTERACTIVITY,” nowissued as U.S. Pat. No. 8,782,713, which is a continuation of U.S.patent application Ser. No. 12/586,057, filed Sep. 16, 2009, titled“SYSTEMS AND METHODS FOR MULTIMEDIA RENDERING, COMPOSITION, ANDUSER-INTERACTIVITY,” now issued as U.S. Pat. No. 8,363,716, which claimsthe benefit of U.S. Provisional Application No. 61/192,136, filed Sep.16, 2008, titled “SYSTEMS AND METHODS FOR ADVANCED INTERACTIVEVIDEO/MULTIMEDIA,” each of which is hereby incorporated by referenceherein in its entirety.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A depicts one example of video content comprising content-awaremetadata;

FIG. 1B is a block diagram of one embodiment of an apparatus forembedding content-aware metadata into a video asset;

FIG. 1C is a block diagram of one embodiment of an apparatus fordecoding a video asset comprising content-aware metadata;

FIG. 2 is a block diagram of one embodiment of an apparatus forauthoring an interactive video/multimedia application;

FIG. 3 is a block diagram of one embodiment of an apparatus configuredto implement an interactive video/multimedia presentation;

FIG. 4 is a block diagram of another embodiment of an apparatus forauthoring an interactive video/multimedia application;

FIG. 5 is a block diagram of another embodiment of an apparatusconfigured to implement an interactive video/multimedia presentation;

FIG. 6 is a block diagram of one embodiment of a system for distributingan interactive video/multimedia application in a distributed network;

FIG. 7 is a diagram of one example of an interactive video/multimediadescription data structure;

FIG. 8 is a diagram of interactive video/multimedia description dataembedded within video content;

FIGS. 9A and 9B depict screen shots of exemplary interactivevideo/multimedia presentations comprising a video-wall-like playback offour and two video assets of the same scene;

FIGS. 10A, 10B, and 10C depict screen shots of an exemplary interactivevideo/multimedia presentation concurrently displaying three out of tenvideo scalability layers;

FIG. 11 depicts screen shots of an exemplary interactivevideo/multimedia presentation comprising a chapter navigationuser-interaction element;

FIGS. 12A, 12B, and 12C depict screen shots of exemplary interactivevideo/multimedia presentations comprising interactive video objects; and

FIGS. 13A, 13B, and 13C depict screen shots of an exemplary interactivevideo/multimedia presentations comprising interactive advertising.

DETAILED DESCRIPTION

Video or other multimedia assets may be “tightly coupled” to informationthat describes the asset (referred to herein as “content-aware”metadata). As used herein, content-aware metadata may refer toinformation that describes or identifies a video content feature,including, but not limited to: a region of interest (ROI) within thevideo content (e.g., a particular portion or encoding aspect of a videodisplay region, one or more video frames, etc.), an object within thecontent (e.g., shape, color region, etc.), motion characteristics ofvideo objects (e.g., local motion vectors of a shape within the videocontent), regions having a particular color, text displayed within orotherwise obtained from the video (e.g., as a graphical element of thevideo, as sub-title information, menu items, text obtained from or byprocessing an audio track, or the like), shape, edge or otheridentifying characteristic, scene change information, scene fingerprint,scene complexity, identification of background and foreground regions,texture descriptors, and the like. In some embodiments, featuresidentified and/or described by content-aware metadata may be generallyreferred to as a ROI and/or an “object.” Content-aware metadata may beembodied as text data (e.g., UTF-8 encoded strings), formatted data(e.g., XML), compressed binary data, or the like.

The content-aware metadata may be tightly-coupled to the video content.As used herein, a tight coupling between content-aware metadata andvideo content may refer to association between metadata and an encodedand/or compressed video bitstream. The association may be at theencoding and/or compression level and may be maintained throughrendering and/or playback. In some embodiments, content-aware metadatamay be embedded within a bitstream comprising the encoded video content(e.g., according to a bitstream syntax). The content-aware metadata maybe tightly-coupled to the video content in other ways. In someembodiments, a video codec or video bitstream may be adapted to includecontent-aware metadata. One example of such a codec is the NextGeneration Video (NGV) codec developed by RealNetworks, Inc., which mayuse content analysis to improve content encoding, compression,scalability, and user-interaction. Information determined using contentanalysis (e.g., shape detection, motion detection, etc.) may beleveraged to produce content-aware metadata describing the encoded videocontent. The production of content-aware metadata may be driven byautomated processes (e.g., machine vision, metadata coding conditions(discussed below), or the like) and/or may be driven by humaninteraction (user-selected and/or identified content-aware metadata).

The content-aware metadata may be embedded into video content as thevideo is encoded and/or compressed. The content-aware metadata may beextracted as the video content is subsequently decoded and/ordecompressed for playback. Since the metadata is tightly-coupled withthe video content at the encoding/compression level, the tightassociation therebetween may be maintained regardless of the mechanismused to playback the encoded video (e.g., independent of network lag,buffering requirements, user interactions, playback performance, playerprocessing resources, and the like). Similarly, the tight-coupling maybe maintained despite changes to the playback caused by user interactionor the like (e.g., pausing video playback, fast forward, chapter skip,channel change, etc.). The tight-coupling described above may allowcontent-aware metadata to be associated with the video content at theframe, scene, and/or video object level.

The tight coupling may be maintained by the renderer used to prepare thedecoded video for playback. The renderer (such as the IVM ApplicationComponents Renderer and Compositor 365 and/or 565 discussed below) maymaintain the synchronism between the decoding and/or decompression ofthe encoded video content and the extracted content-aware metadata. Thesynchronism between the renderer and the decoder and/or content-awaremetadata extraction modules may allow the tight coupling therebetween tobe maintained.

In some embodiments, content-aware metadata may be identified and/orembedded by a video encoder. Encoding and/or compression may includeidentifying features related to the content, such as shapeidentification, motion vectors, scene identification, objectidentification, and so on. These features may be used to encode and/orcompress the video content. The features may also be used ascontent-aware metadata. For example, a video encoder may be configuredto segment a scene or frame of video content into one or more regionlayers. A region layer may be assigned various properties, including,but not limited to: region color, texture, motion, location (within oneor more frames), shape, and the like. Each region may be assigned anidentifier. The region layers (as well as their associated properties)may be made available by the encoder as content-aware metadata.

Alternatively, or in addition, content-aware metadata may be added byanother automated processes (e.g., a process other than a videoencoder), such as, for example, a machine vision process or the like.For example, video content may be processed to automatically identifyhuman faces within the content, identify particular shapes or patternsin a scene (e.g., identify the cars in a scene, edges), track oridentify particular colors, identify and/or parse text appearing in thecontent, identify encoding characteristics of the encoded video andaudio, identify graphical features corresponding to external objects,and so on.

Content-aware metadata may be added by a human operator. For example, ahuman operator may view the video content (e.g., during encoding or apreprocessing step) and identify one or more regions of interesttherein. For example, a human operator may identify a break in theaction of a soccer game or an “offensive” scene (e.g., a scene showingexcessive violence, sexual content, or the like). The operator may causecontent-aware metadata identifying the ROI (e.g., identifying one ormore frames that mark the beginning and/or end ROI in the video) to beembedded within the encoded video content. As will be described below,the operator (or other entity) may reference the embedded content-awaremetadata to control the behavior of an interactive video/multimediaapplication (discussed below). For example, during the identified breakin the action of a soccer game, the user may cause an advertising imageto be displayed (one example of such an advertising multimedia elementis described below in conjunction with FIGS. 13A-13C). In anotherexample, the user may cause an offensive scene in the video (asidentified by the embedded content-aware metadata) to be skipped, and soon.

FIG. 1A depicts one example of encoded video content comprisingcontent-aware metadata. The encoded video content 100 may include datarepresenting a plurality of video scenes or segments 101, 102, and 103.During playback, each segment 101, 102, and 103 may result in thedisplay of a plurality of video frames. The encoded video content 100may include content-aware metadata describing the segments 101, 102,and/or 103. The content-aware metadata may be embedded within theencoded video content 100 (e.g., included as part of the video contentbitstream or other data structure embodying the encoded video content100 ). Accordingly, the content-aware metadata (metadata entries 111,113, 115, 117, 121, 131, 133, and 135, the “FIG. 1A metadata” hereafter)may be tightly coupled to the encoded video 100 to provide formetadata-to-content synchronization that is independent of playbackperformance (e.g., buffering, etc.), playback timing, user interaction,and the like.

In the FIG. 1A example, content-aware metadata 111 may identify a regionof interest (e.g., a racecar) within the segment 101. The content-aware,ROI metadata 111 may define an ROI with bounds of the racecar shape;determine the position of the shape within various frames of the segment101, and the like. In some embodiments, the metadata 111 (or additionalcontent-aware metadata 113 ) may specify local motion vector(s) of theROI. In addition, metadata may identify a background region 115, and aforeground region 117 of the segment 101. For scenes comprisingadditional detail, additional background and foreground layers may bespecified (e.g., in metadata describing a z-buffer data structure or thelike). Additional metadata (not shown) may provide additionalinformation about the scene 101, such as the color and/or texture ofvarious scene elements (e.g., the racecar shape), a model of the shape(in 2-D or 3-D), or the like. For example, a 3-D model of the ROI 111may allow additional multimedia elements (e.g., advertising images) tobe properly overlaid on the racecar identified thereby.

Content-aware metadata 121 may specify an ROI within the scene 102. TheROI metadata 121 may correspond to a shape (not shown) or other featureof the segment 102 (e.g., a foreground region, a background region, acolor region, or the like). The ROI metadata 121 may be determined usingan automated, machine-learning technique, such as pattern matching,template matching, optical character recognition (OCR), or the like.Alternatively, or in addition, video encoding and compression techniquesmay be used, such as shape identification, motion vector estimation, andthe like. As will be discussed below, an ROI may be selectable by a user(e.g., as a hotspot), to allow for user interaction with the segment102. The ROI metadata 121 may include bounding, position, and/or localmotion information.

In the segment 103, content-aware metadata 131 may identify a ROIcomprising one or more encoded video frames within the segment 103. Asused in this context, the ROI metadata 131 may specify a time segment ofthe video content and/or a particular set of video frames as opposed toa region within a video display area. As will be discussed below, time-or frame-related ROI content-aware metadata may be used to triggerevents, such as the display of additional multimedia elements and/oruser-interaction components (e.g., trigger the display of additionalmultimedia elements, such as text overlays, advertising images, audiocontent, and the like).

The frames specified by the ROI metadata 131 may correspond to a scenechange (e.g., as identified by a compression, encoding, or machinevision operation), a scene fingerprint, or the like. The content-awaremetadata 131 may identify a particular set of frames that may be ofinterest in video content. For example, the ROI identified by themetadata 131 may correspond to advertising insertion (e.g., during alull in the action of a baseball game, such as the seventh inningstretch), to a particular event occurring in the content (e.g., a playerhitting a home run), the entrance of a particular player onto the field,the presence of an object within the field of view (such as abillboard), an offensive scene, or the like. In the FIG. 1A example, theROI identified by the content-aware metadata 131 identify the time aparticular player enters the segment 103. The player may have beenidentified by an automated technique (e.g., by identifying a color ofthe player's uniform, uniform markings, such as number and/or name,recognizing facial or other features of the player, and so on). Thesegment 103 may further include content-aware metadata 133 describingthe shape and/or position of the player and/or metadata 135 describingthe motion characteristics of the player.

As discussed above, the FIG. 1A metadata may be encoded with the videocontent 100 (e.g., may be included in the data structure or streamcomprising the video 100 ). Therefore, the content-aware metadata may betightly coupled to the video stream 100 (coupled to the scenes 101, 102,103 and/or the frames and/or objects therein) regardless of thetransport and/or playback mechanism used to display the encoded videocontent 100.

FIG. 1B is a block diagram of one embodiment of an apparatus configuredto encode video content comprising content-aware metadata. The apparatus104 may be implemented on or in conjunction with one or more computingdevices 107. Although not depicted in FIG. 1B, the computing device 107may comprise a processor, memory, computer-readable storage media,communications interface, input/output devices, and the like.

In some embodiments, the apparatus 104 may be configured to identify andembed content-aware metadata into encoded video content 146. In the FIG.1B example, a Content Adaptive Preprocessor 150 may be configured toanalyze video content 140 for encoding by a Content Adaptive Encoder155. The Content Adaptive Preprocessor 150 may receive content codingconditions 142, which may specify input resolution, bitrate, frame rate,application dependent quality expectations, quality of serviceparameters, and the like. The Content Adaptive Preprocessor 150 may usethe content coding conditions 142 to determine preprocessing needed forthe Content Adaptive Encoder 155. The Content Adaptive Preprocessor 150may also receive metadata coding conditions 144, which, as will bediscussed below, may be used by the preprocessor to identifycontent-aware metadata within the video content 140. In someembodiments, the content coding conditions 142 and/or the metadatacoding conditions 144 may be provided and/or modified by an author 105.

The Content Adaptive Preprocessor 150 may analyze the video content 140using video compression and/or machine vision techniques, each of whichmay result in the identification of content-aware metadata. The videocompression information may include scene change information, scenecomplexity information, shape identification, local motion vectors,texture descriptors, region of interest, and the like. In one example,the Content Adaptive Preprocessor 150 and/or the Content AdaptiveEncoder 155 may segment video frames and/or scenes into a plurality ofregion layers, each of which may be assigned respective properties(e.g., color, texture, motion, shape, etc.). The level of detail of thevideo compression information determined by the Content AdaptivePreprocessor 150 may be determined according to the content codingconditions 142 and/or the metadata coding conditions 144. The videocompression information determined by the Content Adaptive Preprocessor150 may be used by the Content Adaptive Encoder module 155 to encodeand/or compress the video content 140. In addition, the videocompression information may be included in the encoded video asset ascontent-aware metadata (e.g., shape and/or color identifiers, motioninformation, foreground, background identification, and the like).

The Content Adaptive Preprocessor 150 may be further configured toanalyze the video content using one or more automated processingtechniques, such as pattern matching, template matching, OCR, faceidentification, and the like. In some embodiments, the machine visiontechniques may be seeded by the image compression information. Forexample, the machine vision techniques may be used to classify a shapeidentified by an image compression technique (e.g., identify the shapeas a racecar and/or as a particular racecar, as a baseball player, andso on). In some embodiments, the machine vision techniques may besupervised by the content metadata coding conditions 144, which mayspecify how metadata is to be identified. For example, the conditionsmay specify that human faces in the content be identified, may specifyhow particular types of objects are to be identified (e.g., how toidentify a racecar using a pattern template or other specifier), mayspecify how specific objects are to be identified (e.g., how to identifymembers of a particular team by uniform characteristics, team name, orthe like), and so on.

Alternatively, or in addition, content-aware metadata may be identifiedand/or specified by an author 105, who may manually identify shapes, orother regions of interest, within the video content 140. The author 105may view the video content 140 as it is being encoded. During theviewing, the author 105 may identify one or more regions of interest (orother features) within the video content. For example, the author 105may identify shapes, or other elements within the video content 140 byselecting shapes or other elements identified by the Content AdaptivePreprocessor 150 or another automated processes. Content-aware metadataidentifying the regions of interest selected by the author 105 may beembedded within the encoded video by the Content Adaptive Encoder 155.

The Content Adaptive Encoder 155 receives the video content 140, theencoding and compression data, and the content-aware metadata identifiedby the Content Adaptive Preprocessor 150. The in FIG. 1 B example, theContent Adaptive Encoder 155 may include a Content Aware Video Encoder157 and a Content-Aware Metadata Encoder 159. The Content Adaptive VideoEncoder 157 may compress and/or encode the video content 140 accordingto the content coding conditions 142. In some embodiments, the ContentAdaptive Video Encoder 157 may provide one or more alternativeencoding(s), which may comprise alternative encodings of the entirevideo content 140 and/or portions thereof (e.g., particular regions,segments, or the like). For example, one or more of the alternativeencodings may correspond to an ROI identified within the video content140 (e.g., by the Content Adaptive Preprocessor 150, the metadata codingconditions 142, the author 105, or the like). The ROI may be provided ina higher resolution and/or bitrate than the rest of the encoded contentto allow a user to “zoom-in” to the specified ROI. In another example,higher-resolution encodings of certain parts of the video content 140may be provided (e.g., as identified by an ROI or the like). Forinstance, the Content Adaptive Video Encoder 157 may generate ahigher-resolution encoding of a goal scored in a soccer match, an autoracing crash, or the like. In an embodiment, the Content Adaptive VideoEncoder 157 may continuously provide a higher resolution encoding, whichhigher resolution encoding or a portion thereof is only stored beyond aspecified period if the author 105 indicates that it should be.

The compressing and/or encoding implemented by the Content AdaptiveVideo Encoder 157 may yield additional content-aware metadata, such asglobal and local motion parameters, region/object shape actuallyencoded, texture descriptors, etc. In some embodiments, the ContentAdaptive Video Encoder 157 may receive the metadata encoding conditions144 to identify content-aware metadata for inclusion in the encodedcontent asset.

The Content-Aware Metadata Encoder 159 may be configured to embed thecontent-aware metadata in the encoded video content 146 (e.g., embed thecontent-aware metadata in the video bitstream). The embedding mayprovide for a tight coupling of the encoded video content and thecontent-aware metadata that is independent of the mechanism used toplayback and/or transport the encoded content 146. In some embodiments,the content-aware metadata may be embedded within a bitsream comprisingthe encoded video content 146 (e.g., according to the bitstream syntaxof the encoding). In some embodiments, the codec used to encode thevideo asset may be adapted to include embedded, content-aware metadata.As discussed above, the NGV codec developed by RealNetworks, Inc. is oneexample of a video codec configured to embed tightly-coupled,content-aware metadata in encoded and/or compressed video content.

In some embodiments, and as shown in FIG. 1B, the content-aware metadatamay be made available as a separate stream or data structure (as encodeddata and/or text). The separate content-aware metadata may flow to anindexing and/or search component for use in indexing and/or classifyingthe encoded video content 146. For example, text content-aware metadata(e.g., identified by an automated machine vision and/or OCR process),may be used to provide for searching within the encoded video content146. For example, content-aware metadata may identify the names of theplayers appearing in a baseball game (e.g., using captioninginformation, uniform labeling, or the like). The content-aware metadatamay be used to allow a user or another application or process to searchfor video content in which a particular player appears. Other metadata,such as color information, shape information, movement, etc. may be usedto classify and/or categorize the encoded content 146. For instance,video content comprising a particular shape (e.g., a racecar) may beindexed and/or classified as “racing related,” video content comprisinga large number of motion vectors may be identified as “action” related,video content having predominantly dark scenes may be identified as“dark,” and so on.

FIG. 1C is a block diagram of an apparatus 108 configured to decodeencoded video content comprising content-aware metadata. The apparatus108 may be implemented on and/or in conjunction with a computing device107 which, as discussed above, may comprise a processor, memory,computer-readable media, communications interfaces, a human-machineinterface (HMI) 177, and the like.

A Content Adaptive Decoder 193 may receive encoded video content 146. AContent Adaptive Video Decoder 195 may be configured to decode and/ordecompress the encoded video content 146 for rendering and/or playback.During the decoding, the Content-Aware Metadata Decoder 197 may extractcontent-aware metadata embedded within the encoded video content 146(e.g., embedded within the bitstream comprising the encoded videocontent 146). Since the extraction of content-aware metadata occursduring decoding and/or decompression, the extracted content-awaremetadata may be synchronized (tightly coupled) to the decoded video(e.g., as a particular frame or scene is decoded, content-aware metadatadescribing the frame or scene is extracted). The synchronism between thecontent-aware metadata and the video frames and/or scenes describedthereby may be maintained regardless of the mechanism used to playback,transmit, and/or interact with the encoded video. As discussed above,the tight coupling may be maintained by the use of synchronizedrendering and/or playback components (e.g., modules 365 and/or 565described below).

The decoded and/or decompressed video content may flow to a ContentAdaptive Postprocessor 198, which may be configured to selectappropriate playback parameters for the decoded video (e.g., videoresolution, bitrate, etc.). The Content Adaptive Postprocessor 198 mayselect the playback parameters according the capabilities of the displaydevice on which the video is to be displayed (e.g., the display deviceavailable on the HMI 177). The Content Adaptive Postprocessor 198 may beconfigured to output alternative bitrate(s) and/or resolution(s)depending upon the capabilities of the HMI 177 (e.g., for picture inpicture, zoom, alternative views, etc.).

The extracted content-aware metadata may flow into an IVM applicationfor rendering and/or playback (as shown in FIG. 5). Alternatively, or inaddition, the extracted content-aware metadata may flow to an indexingservice or datastore which may use content-aware metadata to indexand/or classify the encoded video content 146 as described above.

The content-aware metadata discussed above may be leveraged to enableinteractive video/multimedia applications (“IVM applications”hereafter). An IVM application may be described by interactivevideo/multimedia application description data (“IVMA data” hereafter).The IVMA data may define the rendering, composition, anduser-interactivity features of the IVM application. The rendering,composition, and/or user-interactivity features may be tied tocontent-aware metadata. For example, the content-aware metadata maydefine spatio-temporal relationships between multimedia elements in theIVM applications, may define and/or trigger user interactivity features,and so on.

An IVM application may comprise multiple, interactive video/multimediaelements including, but not limited to: video content, audio content,graphical content (e.g., text, images, rendered 2 D and/or 3 D graphics,etc.), and the like. The multimedia content may be rendered and/orcombined in the IVM application in various ways (as specified by theIVMA data). For example, an IVM application may combine multiple contentassets using transparency (e.g., an image may be transparently overlaidon a video object), depth mapping (e.g., z-buffering), alpha-blending,clipping, audio mixing, motion animation, and the like. The IVMapplication may provide for user interaction with various elementsdefined in the IVM application (e.g., user interaction with regions ofinterest of video content, objects defined in the video content, userinterface components, and the like).

FIG. 2 is a block diagram of one embodiment of an apparatus 200 forauthoring an IVM application. The modules shown in FIG. 2 may beimplemented using one or more computing devices 207, which, as discussedabove, may comprise a processor, memory, computer-readable media,communications interfaces, input/output devices, and the like. Thecomputing device(s) 207 used to implement the apparatus 200 may beconfigured to operate in particular ways and/or perform particularmethods by instructions stored on a computer-readable storage medium(not shown).

The author 105 may author an IVM application using an IVM applicationdescription framework or language, which may be defined in a schema 220.The schema 220 may define IVMA data elements, element attributes, andelement relationships.

The RMI schema is one example of a framework or language for developingIVM applications. The RMI schema may define the rendering, composition,and interactivity behavior of an IVM application. Related, implementingtechnologies (disclosed herein) may provide for encoding, decoding,transporting, and presenting an IVM application authored according tothe RMI schema. Although the RMI schema is discussed herein, theteachings of this disclosure are not limited in this regard and could beadapted to use any schema or framework capable of defining the behaviorof an IVM application (e.g., capable of defining rendering, composition,and/or user-interactivity features of and IVM application). Moreover,although the RMI schema discussed herein is implemented as an XMLschema, the elements, attributes, and/or element relationships definedin the RMI schema could be adapted to use any language definition and/orapplication framework known in the art.

Using the Interactive Video/Multimedia Application (IVMA) Authoringmodule 215, the author 105 may define an IVM application (according tothe RMI schema). The IVM application may be defined as IVMA data, whichmay specify the content assets to be included in the IVM application(e.g., as links or paths), define how the content assets are to berendered and/or composed, define relationships between the contentassets (e.g., specify composition, layering, animation (in JavaScript orECMAScript, etc.)), define user-interactivity features, and so on. Therelationships between the content assets and/or user interactionsdefined in IVMA data may be predicated upon the content-aware metadataencoded within one or more of the content assets (e.g., within anencoded video asset).

The IVMA data may be authored using a text editor (not shown).Alternatively, or in addition, the IVMA Authoring module 215 may includea graphical authoring environment configured to produce IVMA data,including, but not limited to: a text editor, a graphical userinterface, an integrated development environment (IDE), and the like.

The author 105 may select media assets for inclusion in the IVMapplication from the media asset information datastore 210.Alternatively, or in addition, media assets may be selected from other,network-accessible media asset information datastores, such as theremote media asset datastore 268, which is accessible on the network 250via a communications interface module 267. The IVMA data may refer tothe selected media assets by links or paths (e.g., XPath, URL, URI,etc.).

In some embodiments, the apparatus 200 may include media decoders 270,which may be configured to decode the selected media assets for playbackto the author 105, allowing the author 105 to evaluate media assetsconsidered for inclusion in the IVM application. One or more of themedia decoder(s) 270 may be configured to extract content-aware metadatafrom encoded media assets.

One or more of the selected media assets may be an encoded video asset,which may include content-aware metadata. In some embodiments, thecontent-aware metadata may be referenced in the IVMA data to synchronizeand/or control the behavior of the IVM application. The encoded videoasset may have been generated (e.g., encoded and/or compressed) usingthe apparatus 104 described above (using the Content AdaptivePreprocessor 150 and/or Content Adaptive Encoder 155 (including theContent Adaptive Video Encoder 157 and/or the Content-Aware metadataEncoder 159 ) of FIG. 1B). The metadata may be available in the mediaasset information datastore 210. Alternatively, or in addition, theauthor 105 may have access to a content adaptive preprocessor (notshown) and/or Content Adaptive Video Encoder (not shown), which may beused to encode video comprising content-aware metadata during IVMapplication authoring. One example of such a system is described belowin conjunction with FIG. 4.

The author 105 may key the rendering, composition, and/oruser-interactivity features of the IVM application to one or morecontent-aware metadata entries embedded within an encoded video asset.For example, content-aware metadata may be used to define the temporalcombination of multimedia assets. In one example, an encoded video assetmay include content-aware metadata identifying an ROI within the video,which may correspond to a break in the action of a baseball game (e.g.,the seventh inning stretch). The author 105 may want the IVM applicationto display an advertising image (or other multimedia element) during thebreak. See FIGS. 13A-13C. The author 105 may define this behavior bykeying the display of the advertising images to the ROI content-awaremetadata in the video (e.g., the IVMA data defining the IVM applicationmay key the display of the advertising images to the content-awaremetadata). The IVMA data may reference the ROI metadata using anidentifier assigned to the ROI metadata in the encoded video asset,which may comprise an alpha numeric identifier, a link, a path (e.g.,XPath, URL, URI), or the like. As will be discussed below, as theencoded video content is decoded for playback in the IVA application,the decoder may inform the IVA application that the identifiedcontent-aware metadata has been detected. Responsive to detection of thecontent-aware metadata, the IVA application may cause the video scene tobe updated (e.g., cause the advertising images to be displayed).

Content-aware metadata may also be used to define the spatialrelationships between multimedia assets in an IVM application. Forexample, content-aware metadata within an encoded video asset mayspecify a spatial region (ROI) within one or more video frames and/orscenes. The region of interest may correspond to an object within thevideo (e.g., a baseball player, racecar, or the like), a billboard, auser-defined area, a color region, or the like. The author 105 may wantthe IVM application to display another element in the same area orsimilar area as the ROI in the video scene (e.g., at an offset from theROI). In one example, the author 105 may want the IVM application todisplay a graphic showing the batting average of a baseball player whenthe player appears in the video (the position of the baseball player maybe defined by the content-aware metadata). The author 105 may want thegraphic to be displayed next to the player as he moves within the field.The author 105 may define this behavior by keying the display locationof the multimedia element (e.g., the batting average graphic) to thecontent-aware metadata in the encoded video (e.g., the IVMA datadefining the IVM application may key the display location of theelement(s) to the location specified by the content-aware metadata). Theauthor 105 may cause the display area to move with the player byreferencing other content-aware metadata, such as local motion vectors,or the like.

The author 105 may leverage content-aware metadata to define otheraspects of the IVM application (aside from spatio-temporal combinationexamples discussed above.). For example, and as discussed above,content-aware metadata may be used to determine how multimedia elementsare to be combined. In one example, the content-aware metadata mayprovide depth-mapping information (e.g., may identify foreground andbackground regions within a scene), which may be used to combine themedia assets (e.g., using a z-buffer, alpha blending, or otherdepth-aware combination technique). In another example, content-awaremetadata may define a model (in 2-D or 3-D) of an object displayedwithin one or more frames. The model may be used to incorporate othermultimedia elements into the scene (e.g., translate an advertising imageinto the 3-D model of the object to overlay the image thereon, to carrycontrols for the display, etc.).

In some embodiments, user-interactivity features of the IVM applicationmay be enabled by content-aware metadata. For example, content-awaremetadata may be used to define a user-selectable hotspot (discussedbelow) corresponding to a region of interest and/or objects within oneor more video frames or scenes. Similarly, user-interactivity featuresmay be predicated upon the occurrence of particular events in the video(e.g., using ROI content-aware metadata), scene changes, scenefingerprint, scene movement, object and object movement within thescene, and the like.

In one example, the author 105 may designate an ROI (or other object)defined in content-aware metadata as a user-selectable hotspot. The IVMAdata may specify one or more scene changes to occur responsive to userselection of the hotspot. For example, selection of a “racecar” regionof interest hotspot may cause an additional multimedia elementcomprising information about the racecar to be displayed with the videoasset (e.g., the driver of the car, the number of laps since the car'slast pit stop, and the like). As discussed above, the IVA applicationmay use the content-aware metadata to cause the display to be placed inthe vicinity of the racecar object and/or may cause the display tofollow the movement of the racecar as it moves around the track (e.g.,using content-aware metadata identifying the position and/or localmotion vectors of the object).

In another example, the IVMA data may specify that selection of ahotspot is to cause an enhanced version (e.g., zoomed-in view) of theROI to be displayed in the IVM application. For instance, selection ofthe racecar hotspot may cause a region around the racecar to bepresented in a higher resolution (e.g., overlaid on the first contentasset, presented in a picture-in-picture view, or the like). In otherexamples, a user-controlled object may change its position according toglobal or local motion vectors available as content-aware metadata inthe encoded video, selection of a table of contents multimedia elementmay cause playback to skip to another chapter, and so on. In otherexamples, the IVM data may specify that a hotspot, when clicked on orotherwise when interacted with by the user, may activate a link (e.g.,XPath, URL, URI) to other data.

Although particular types of content-aware metadata enabled renderings,compositions, and/or user-interactivity features are described herein,the disclosure is not limited in this regard. The teachings of thisdisclosure could be adapted to enable other types of rendering,composition, and/or user-interactivity features, including, but notlimited to: combination of plural multimedia assets, overlay ofadvertising imagery, overlay of graphical elements (e.g., text, 2Dgraphical elements, 3D graphical elements, vector graphics, and thelike), audio mixing, zooming, panning, object manipulation,user-interface elements, and the like.

As discussed above, the author 105 may define an IVM application as IVMAdata, which may identify the media assets to include in the IVMapplication, specify how the assets are to be rendered and composited,and define any user-interactivity features of the IVM application. Theauthor 105 may author the IVMA data using a separate text editor (notshown) and/or may author the IVMA data using the IVMA Authoring module215.

The IVMA data may be embodied as XML data that complies with the schema220 (e.g., the RMI schema). The IVMA Authoring module 215 may beconfigured to validate authored IVMA data against the schema 220. Thevalidation may comprise validating the paths or links to content assets(e.g., media asset information datastore 210 or network storage location(not shown)) and/or validating references to content-aware metadata inthe IVMA data (e.g., validating the content-aware metadata referenced inthe IVMA data exists in the corresponding encoded video assets).

The textual representation of the IVMA data may optionally flow to anIVMA Data Parser module 230, which may be configured to parse the IVMAdata into two or more chunks. The chunks may flow to an IVMA DataEncoder module 235, which may be configured to encode the IVMA datachunks into a compressed, binary format. In some embodiments, however,the IVMA Parser 230 and/or IVMA Data Encoder 235 module(s) may bebypassed, and the IVMA data may remain as text data and/or as a singleset of binary encoded data (e.g., the textual representation may flowdirectly to the encoding module 235 ). Although not depicted in FIG. 2,in some embodiments, the apparatus 200 may include switches,multiplexers, or other data flow components configured to route the IVMAdata to/from the IVMA Data Parser 230 and/or IVMA Data Encoder 235modules.

The IVMA data produced by the apparatus 200 may flow to a playbackcomponent which may provide the enabling technologies required toimplement the IVM application defined thereby (e.g., implementrendering, composition, and/or user-interactivity features defined bythe IVMA data). FIG. 3 shows a block diagram of one embodiment ofapparatus 300 configured to implement an IVM application defined by IVMAdata. The apparatus 300 may be implemented using a computing device 307,which may comprise a processor (not shown), memory, computer-readablestorage media (e.g., IVMA data storage 364, schema 320, local assetstorage 366 and the like), input/output devices (e.g., HMI 377 ),communications interfaces (e.g., communication interface 367 ), and thelike.

IVMA data 340 may be received by an IVMA Data Decoder and ChunksCollector module 360, which may be configured to decode and/orreconstruct chunks of the IVMA data 340 (e.g., aggregate one or morechunks of the IVMA data 340 in the proper order). The decoded and/orreconstructed IVMA data may flow to the IVMA Data Interpreter and AssetRequest Generator module 362. In some embodiments, the encoded and/orreconstructed IVMA data may be stored in the IVM application datastore364 for subsequent access.

The IVM Application Interpreter and Asset Request Generator module 362may interpret the IVMA data. The interpretation may comprise accessing aschema 320, such as the RMI schema discussed above. If the IVMA data 340is embodied as binary encoded data, the schema access may not berequired. Alternatively, or in addition, the IVM Application Interpreterand Asset Request Generator module 362 may access previously stored IVMAdata directly from the IVM application datastore 364. For example, theIVMA data 340 may be downloaded and/or transmitted to the apparatus 300before playback occurs (e.g., as part of a Really Simple Syndication(RSS) feed, a dedicated download, or the like). The IVMA data 340 may bestored in an IVM application datastore 364, until a user elects toplayback the IVM application defined thereby, at which point the IVMAdata 340 may be accessed.

As discussed above, IVMA data may reference one or more multimediaassets for playback in the IVM application. The IVM applicationinterpreter and access request generator module 362 may access contentassets referenced in the IVMA data from a local file system 366. If thecontent assets referenced in the IVMA data are not available locally (orupdated versions are available), the content assets may be accessed fromthe remote media asset datastore 368 via the network 350 (e.g., TCP/IPnetwork, Internet, WAN, LAN, or the like) using a communicationsinterface 367 (e.g., Ethernet interface, wireless interface, or thelike).

During playback, content assets that are in raw digital form (PCM) mayflow directly to an IVM Application Renderer and Composer module 365.Content assets that are available in an encoded, compressed format maybe decoded by an appropriate media decoder module 370 and/or 372. In theFIG. 3 example, the media decoders 370 may be configured to decode mediaassets available on the local file system 366, and the media decoders372 may be used to decode media assets accessed from the remote mediaasset datastore 368. The media assets may be decoded into PCM form foruse by the IVM Application Renderer and Compositor module 365. In someembodiments, the media decoder modules 370 and/or 372 may be configuredto extract content-aware metadata embedded within encoded multimediaassets.

Content-aware metadata embedded within encoded video assets referencedin the IVMA data may be extracted by the media decoders 370 and/or 372during playback. The content-aware metadata may be provided to the IVMApplication Interpreter and Asset Request Generator module 362, which,as discussed above, may use the content-aware metadata to update the IVMapplication (e.g., by a direct communication path therebetween (notshown) and/or through a communication path provided by the IVMApplication Components Renderer and Compositor module 365 ). Asdiscussed above, the content-aware metadata may be used by the IVMApplication Interpreter and Asset Request Generator module 362 todetermine rendering, composition, and/or user-interactivity features ofthe IVM application as defined by the IVMA data.

The IVM Application Interpreter and Asset Request Generator module 362may interpret the IVMA data and extracted content-aware metadata todetermine how the media assets referenced by the IVMA data should berendered and composed in the IVM Application. The IVM ApplicationInterpreter and Asset Request Generator module 362 may provideinstructions to the IVM Application Components Renderer and Compositormodule 365 on how the media assets are to be rendered, animate, and/orcomposite the media assets per the IVM Application.

The output of the IVM Application Components Renderer and Compositor 365flows to the IVMA Player Application 375, which may be configured tocause the HMI 377 to present the rendered IVM application to a user 380.In some embodiments, the IVMA Player Application 375 may be embeddedwithin another application, such as a plug-in of a web browser. The HMI377 may include a visual display (e.g., monitor), audio outputs (e.g.,amplifier, speakers, etc.), input devices (e.g., a remote control,keyboard, mouse, touchpad, touchscreen, motion sensor, camera,microphone, etc.), and the like.

The HMI 377 may receive inputs from the user 380 via various inputdevices. The user inputs may include, but are not limited to: a mouseinteraction (e.g., movement of a mouse cursor on a display, clicking amouse button, or the like), a touch pad interaction, a touch screeninteraction, a motion sensor interaction (e.g., tilting the HMI 377 or acomponent thereof), a keyboard input, an audio input (e.g., via amicrophone), a visual input (e.g., via a camera), or the like. The userinteraction may be represented as a user-interaction event 382, whichmay be fed back through the IVMA Player Application 375 and/or the IVMApplication Components Renderer and Compositor module 365 to the IVMApplication Interpreter and Asset Request generator 362, which mayinterpret the user-interaction event 382 according to user-interactivityfeatures defined in the IVM application. The user-interactions may causea change in behavior of certain elements of the IVM application (e.g.,cause certain elements to be added and/or removed, change thepositioning of elements, and the like). Responsive to the userinteraction(s), the IVM Application Interpreter and Asset RequestGenerator 362 may update the IVM application (e.g., update the scenedisplayed to the user 380 ), which may comprise accessing new mediaassets (e.g., from the local 366 and/or remote 368 content assetdatastores), updating the rendering a composition instructions providedto the IVM Application Components Renderer and Compositor module 365,and the like.

FIG. 4 shows another embodiment of an apparatus for authoring IVMapplication. The apparatus 400 may be implemented on a computing device407, which may comprise a processor, memory, computer-readable media,input/output devices, and the like. The apparatus 400 includes a mediaasset information datastore 410, IVM

Authoring module 415, schema 420, IVMA Data Parser module 430, and IVMAData Encoder module 435, which may operate similarly to the modules 210,215, 220, 230, and 235 described above in conjunction with FIG. 2. Theapparatus 400 also includes a Content Adaptive Preprocessor 450 and aContent Adaptive Encoder 455 (comprising a Content Adaptive VideoEncoder 457 and a Content-Aware Metadata Encoder 459 ), which mayoperate similarly to the modules 150, 155, 157, and 159 described abovein conjunction with FIG. 1B. Although not shown in FIG. 4, the apparatus400 may further include a communication interface communicativelycoupled to a network through which media assets stored in one or moreremote media asset datastore(s) may be accessed. In addition, theapparatus 400 may include one or more media decoders (not shown)configured to provide for playback of encoded media assets, which mayallow the author 105 to evaluate the media assets for inclusion in theauthored IVM application.

In the FIG. 4 example, raw video frames may be received by a contentadaptive preprocessor 450 and the Content Adaptive Video Encoder 455.The preprocessor 450 may receive content coding conditions (not shown)and metadata coding conditions (not shown) from the author 105 via theauthoring module 415. The Content Adaptive Preprocessor and ContentAdaptive Encoding modules 450 and 455 may be configured to encode theraw video content as described above. During the encoding, content-awaremetadata may be identified and embedded within the encoded video contentas described above in conjunction with FIG. 1B (e.g., within a bitstreamcomprising the encoded and/or compressed video content). Thecontent-aware metadata identified and embedded within the encoded videoasset may be made available to the IVMA Authoring module 415.

The apparatus 400 may also receive a compressed and/or encoded videostream 446. The encoded video content 446 may be received by the VideoStream Chunks Parser module 480, which may parse and/or reconstruct thevideo content. Content-aware metadata within the encoded video stream446 (if any) may be provided to the IVMA Authoring module 415 (e.g.,during a reconstruction, decoding, and/or decompression processimplemented by the Video Stream Chunks Parser 480 ). In someembodiments, the reconstructed video content may flow to thepreprocessor 450 and/or Content Adaptive Encoder 455 for re-encodinginto another video codec (e.g., a video codec having a differentbitrate, resolution, or the like) and/or for the identification andaddition of content-aware metadata.

The encoded video content produced by the video stream parser 480 and/orContent Adaptive Encoder 455 may flow to the Multiplexer and Packetizermodule 482, which may combine encoded video content with IVMA dataauthored by the author 105 (in textual or binary, compressed format)into a single stream and/or as a set of distributed IVMA data instances(as defined in the schema 420 ). The IVMA data authored by the author105 (using the IVMA Authoring module 415 ) may be processed by the IVMAData Parser module 430 and/or IVMA Data Encoder modules 435. Asdiscussed above, the IVMA Data Parser module 430 may be configured tosegment the IVMA data into chunks. Each chunk of IVMA data may compriseone or more IVMA data elements or instances (e.g., in accordance withthe schema 420 described below). The chunks may be binary encoded and/orcompressed by the IVMA Data Encoder module 435. Alternatively, the IVMAdata may bypass the IVMA Data Parser module 430 and/or IVMA Data Encodermodule 435, and flow directly to the Multiplexer and Packetizer module482 and/or made available from the apparatus 400 as text.

The Multiplexer and Packetizer module 482 may be configured to combinethe chunked IVMA data (in text or compressed, binary format) with theencoded video content (produced by the Video Stream Chunks Parser 480and/or the Content Adaptive Preprocessor 460 and Content Adaptive VideoEncoder 455 ). The IVMA data chunks may be multiplexed with the encodedvideo content in various different ways, one example is described belowin conjunction with FIGS. 7 and 8. The Multiplexer and Packetizer module482 may be further configured to packetize the multiplexed stream fortransmission on a network. The packet and/or transmission frame size maybe selected according to the performance and/or capabilities of thenetwork infrastructure used to transmit the data. In addition, theapparatus 400 may make separate streams of encoded video and IVMA dataavailable in non-multiplexed form. In some embodiments, thenon-multiplexed video and IVMA data may be separately packetized fornetwork transmission (e.g., by the Multiplexer and Packetizer module 482or another module (not shown)).

FIG. 5 depicts another embodiment of an apparatus 500 configured toimplement an IVM application defined in IVMA data. The apparatus 500 maybe implemented on a computing device 507, which, as discussed above, maycomprise a processor, memory, computer-readable media, input/outputdevices, and the like.

The apparatus 500 may include a schema 520, IVMA Data Decoder and ChunksCollector module 560, IVM Application Interpreter and Asset RequestGenerator module 562, IVM Application datastore 564, IVM ApplicationComponents Renderer and Compositor module 565, local file system 566,communications interface 567, media decoders 570 and/or 572, IVMA PlayerApplication 575, and HMI 577, each of which may operate similarly to thecomponents 360, 362, 364, 365, 366, 367, 370, 375 and 377 describedabove in conjunction with FIG. 3. The apparatus 500 may further includea Content Adaptive Decoder module 593 (including Content Adaptive VideoDecoder 595 and a Content-Aware Metadata Decoder 597 modules), which mayoperate similarly to the modules 193, 195, and/or 197 described above inconjunction with FIG. 1C.

The apparatus 500 may be configured to receive IVMA data as a separatestream (IVMA data stream 540 ) and/or multiplexed with video content(video with IVMA data 542 ). Multiplexed video and IVMA data 542 may bereceived by a DePacketizer and Demultiplexer module 590, which may beconfigured to depacketize and/or demultiplex the IVMA data embeddedwithin the video asset 542. The IVMA data (if any) extracted from thevideo asset 542 may flow to the IVMA Data Decoder and Chunks Collectormodule 560, which may reconstruct the IVMA data for use by the IVMApplication Interpreter and Asset Request Generator module 562.Alternatively, or in addition, the apparatus 500 may receive a separateIVMA data stream 540, which may flow directly IVMA Data Decoder andChunks Collector module 560.

The encoded video (with the IVMA data having been extracted therefrom)may flow to the Content Adaptive Decoder module 593 for decompressionand/or decoding (by the Content Adaptive Video Decoder module 595 ).During decoding and/or decompression, the Content-Aware Metadata Decoder597 may extract content-aware metadata from the encoded video content.The extracted content-aware metadata may flow to the IVM ApplicationInterpreter and Asset Request Generator 562.

The decoded video content (output by the Content Adaptive Video Decoder595 ) may be prepared for rendering and/or composition by the ContentAdaptive Postprocessor 598, which may be configured to selectappropriate playback parameters for the decoded video content (e.g.,resolution, bitrate, etc.). The playback parameters may be determinedaccording to the HMI 577 on which the IVM application is to bedisplayed, according to the processing resources available, and thelike.

As discussed above, during the decoding, the Content-Aware MetadataDecoder 597 may identify and extract content-aware metadata embeddedwithin the encoded video content (e.g., embedded within the bitstream ofthe encoded video content). Alternatively, or in addition, content-awaremetadata, such as local motion vectors, shape identification, and thelike, may be determined on-the-fly as the video asset is decoded by theContent-Aware Video Decoder 595.

The content-aware metadata extracted and/or determined by the ContentAdaptive Decoder module 593 may flow to the IVM Application Interpreterand Asset Request Generator module 562, which may use the content-awaremetadata to provide content asset rendering, composition, anduser-interactivity features of IVM application (as defined in the IVMAdata).

The content-aware metadata extracted by the Content Adaptive Decoder 593and/or the media decoders 570 and 572, may be continually updated asvideo content is decoded for playback in the IVM application.Accordingly, the IVM Application Interpreter and Asset Request Generatormodule 562 may be capable of continually updating the scene renderingand composition instructions provided to the IVM Application ComponentsRenderer and Compositor module 565 responsive to the content-awaremetadata.

FIG. 6 is a block diagram depicting distribution of an IVM applicationin a distributed network environment 600. An IVM application may beimplemented on a number of different client devices 605A-605F,including, but not limited to: a mobile phone (e.g., cell phone, smartphone, Personal Digital Assistant (PDA) etc.) 605A; a Portable MediaPlayer device (PMP) 605B; a mobile computing-device (e.g., laptop,netbook, or the like) 605C; a personal computer (e.g., a desktop) 605D;a Set Top Box (STB) 605E; an network-aware display device (e.g.,Internet connected television) 605; or the like. The client devices605A-605F may be communicatively coupled to one or more servers 610 viaa network 650. The network 650 may include, but is not limited to: adedicated network, a local area network (LAN), a wide area network(WAN), the Internet, a fiber-optic network, a wireless network, a Wi-Finetwork, a mobile cellular network, a wireline home network, a publicswitched telephone (PSTN) network, a combination of networks, or thelike.

The one or more servers 610 may provide IVMA data defining an IVMapplication to the client devices 605A-605F, each of which may comprisea compatible IVM application playback component. The IVM playbackcomponent may comprise software and/or hardware components.Alternatively, or in addition, the client devices 605A-605F may includemedia player components capable of providing for synchronized mediadecoding and playback. The media player components may be furtherconfigured to extract content-aware metadata embedded within encodedvideo assets.

The servers 610 may provide media assets referenced by the IVMapplication (e.g., as separate streams and/or combined into a single,multiplexed stream with the IVMA data). The servers 610 may serve IVMAdata on request to multiple clients 605A-605F. The IVM application mayreference locally available media assets and/or media assets obtainedfrom the servers 610 (or from other sources) by downloading orstreaming.

In some embodiments, the same set of IVMA data may be either servedunaltered to different client devices 605A-605F and/or may be altered tosuit the processing resources, display capabilities, and/or networkconditions of a particular client device 605A-605F. Alternatively, or inaddition, an IVM application (as defined by IVMA data) may be configuredto provide renderings and/or compositions of media assets according tothe device on which the IVM application is to be implemented. Forexample, the IVM application may specify that high-resolution content bedisplayed on desktop computer 605D and/or STB 605E clients, whereaslower resolution content should be displayed on a mobile phone client605A.

As discussed above, IVMA data may be segmented into two or more chunks,which may be optionally binary encoded and multiplexed into videocontent. FIG. 7 depicts one example of a Unified Modeling Language-(UML-) type representation of segmented IVMA data. In the FIG. 7example, the IVMA data is embodied as RMI presentation data compliantwith the RMI schema disclosed herein.

As shown in FIG. 7, an RMI presentation 700 may be segmented into aplurality of chunks of RMI data 710. Each RMI chunk 710 may comprise oneor more RMI instances 715, each of which may comprise one or more RMIelements (as defined by the RMI schema). In accords with the RMI schema,a properly formed RMI presentation may include one or more required RMIelements 720 and/or more optional RMI elements 720. In the RMI schema,various RMI elements 720 have been given names and attributes to reflecttheir respective properties and behavior.

As defined by the RMI schema, an RMI instance 715 may be embedded withina media asset (for later reconstruction). For example, an RMI instance715 may be embedded within uncompressed and/or compressed digital videostreams (e.g., by multiplexing the RMI instance data within videocontent using a Multiplexer and Packetizer module, such as module 482 ofFIG. 4). Alternatively, or in addition, an RMI instance 715 may beembedded within other content types, such as digital audio streams,texture data (e.g., texture mapping, texture atlas, or other data),streaming modeling data, or the like. In embodiments in which an RMIinstance 715 is placed within video content at a video frame level, theRMI instance 715 may be uniquely identified by a start code or an offsetwithin the video frame. Similar identification may be provided inembodiments in which an RMI instance 715 is embedded within othercontent types.

The following is an example of placement of an RMI instance (e.g., anRMI instance element) within a video stream. The following examples areimplemented using a prior version of the RMI schema (the schema versiondisclosed in U.S. Provisional Application No. 61/192,136, filed Sep. 16,2008, for “Systems and Methods for Advanced InteractiveVideo/Multimedia”) incorporated by reference above. However, thefollowing examples could be adapted to conform to the updated RMI schemadisclosed herein.

<?xml version=“1.0” encoding=“UTF-8”?> <RMIxmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance”xsi:noNamespaceSchemaLocation=“file:///F:/DEMO_F/RMI/2008-09-10-RMIPlayerDX/rmi104.xsd” FrameRate=“24”> <Canvas Width=“1280”Height=“720”> </Canvas> <VideoSource Xpos=“0” Ypos=“40” VideoID=“40”DepthOrder=“1” Width=“704” Height=“352” FrameRate=“23.976”StartPlaying=“false” RenderedWidth=“1280” RenderedHeight=“640”SyncStreamIDs=“140” > <LocalObjectLocation>E:\VIDSRC\ch36_704x352.yuv</LocalObjectLocation> <MediaControlActionControl=“PLAYPAUSE” TargetID=“40”/> </VideoSource> <AudioSourceAudioID=“105” StartPlaying=“false”> <LocalObjectLocation>E:\VIDSRC\ch01.d2v.wav</LocalObjectLocation> </AudioSource> <TextOverlay ID=“205”DepthOrder=“3” Duration=“5000000”> <BoundingBox TopLeftX=“28”TopLeftY=“26” BottomRightX=“40” BottomRightY=“40” /> <Color R=“200”G=“200” B=“200”/> <Font FamilyName=“Chicago” Size=“11” Weight=“bold”/><Text>CHAPTER 1</Text> </TextOverlay> <TextOverlay ID=“255”DepthOrder=“3” Duration=“5000000” Visible=“0”> <BoundingBoxTopLeftX=“1038” TopLeftY=“656” BottomRightX=“1238” BottomRightY=“674” /><Color R=“200” G=“200” B=“200”/>  <Font FamilyName=“Chicago” Size=“13”Weight=“bold”/> <Text>Extras: Making of Movie</Text> </TextOverlay><VideoChain ID=“159” Visible=“1”> <VideoSource Xpos=“256” Ypos=“0”VideoID=“59” DepthOrder=“2” Width=“64” Height=“32” AlphaBlend=“200”FrameRate=“23.976” > <LocalObjectLocation>E:\VIDSRC\ch05_64x32.yuv</LocalObjectLocation> <TargetAction Target ID=“9”SyncWithSource=“true”/> </VideoSource> </VideoChain> <RMIPolygonFramePolygon=“0” FillPolygon=“1” ID=“300” DepthOrder=“3” LineR=“200”LineB=“20” LineG=“20” FillR=“255” FillG=“255” FillB=“255”AlphaBlend=“50” Visible=“0”> <Point X=“205” Y=“120”></Point> <PointX=“305” Y=“120”></Point> <Point X=“355” Y=“220”></Point> <Point X=“305”Y=“320”></Point> <Point X=“205” Y=“320”></Point> <Point X=“155”Y=“220”></Point> <Point X=“205” Y=“120”></Point> <TargetActionTargetID=“80”/> <Animation Duration=“2000” FinalXPos=“600”FinalYPos=“200” /> </RMIPolygon>  </Scene>  </RMIInstance> </RMI>

As discussed above, some RMI elements 720, such as Table of Contents(ToC) element may apply to large portions of the RMI presentation 705(e.g., may be “global” RMI elements). As illustrated above, an RMIpresentation 705 (in binary and/or textual form) may be transmitted as aseparate stream or file and/or may be embedded within another contenttype (e.g., embedded within a video stream). As a user accesses the RMIpresentation 705, the user (or the RMI presentation 705 itself) maycause various different portions of the video stream to be played back(e.g., may navigate to different portions of chapters within the videostream, switch between streams or channels, or the like). If the RMIpresentation 705 (or global RMI elements 720 thereof) is embedded onlywithin a first portion of the video stream, and another portion of thestream is requested, it may be difficult to get access to the RMIpresentation 705 and/or the global elements 720 thereof. Accordingly, insome embodiments, the RMI presentation 705 in whole or in part (e.g.,only “global” RMI elements 720, such as ToC elements), may beperiodically repeated (re-embedded) Alternatively, or in addition, theRMI presentation 705 may be distributed throughout the media stream atparticular, known intervals, such as the beginning or end of a frame(uncompressed or compressed). When one or more RMI elements 720 are soembedded, they may be referred to as RMI instances 715. The embedded RMIinstances 715 may be those instances that are relevant to the proximateframes. In addition, RMI elements 720 considered to be more “global” innature (e.g., having the potential to affect the presentation as awhole, such as ToC elements, or the like) may be included, which maycause the global RMI elements 720 to be repeated.

FIG. 8 depicts one example of video content 800 comprising embedded RMIinstances 815A-815D (each of which may comprise one or more RMI elementsand associated attributes). The location of the RMI instances 815A-815Dwithin the video content 800 may be determined by respective integralnumber video frames 817A-817D. A first RMI instance 815A may provide“global” RMI presentation elements including a table of contents (ToC)and the like. As shown in FIG. 8, multiple RMI instances 815A-815D maybe distributed within the video content 800. In some embodiments,subsequent RMI instances (e.g., RMI instances 815B-815D) may includeglobal RMI elements, such as the ToC element within the RMI instance815A.

The following is a one example of an RMI presentation comprisingembedded RMI instances:

<?xml version=“1.0” encoding=“UTF-8”?> <RMIxmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance”xsi:noNamespaceSchemaLocation=“file:///F:/DEMO_F/RMI/2008-09-10-RMIPlayerDX/rmi104.xsd” AdProfile=“AdProfileNascarDemo4.aprf”FrameRate=“60”> <Canvas Width=“1280” Height=“720”> </Canvas> <!-- Usageof ImageOverlay element RenderedWidth=“176” RenderedHeight=“144” --> <RMIInstance FrameOffset=“1”> <!-- Bud Ad1 --> <Scene ID=“0”> <!-- Mainvideo --> <VideoSource Xpos=“31” Ypos=“28” VideoID=“10” DepthOrder=“1”Width=“640” Height=“360” RenderedWidth=“1214” RenderedHeight=“658”SyncStreamIDs=“20” FrameRate=“59.97” > <LocalObjectLocation>F:\DEMO_F\NGV\Nascar\rac_01_09_40a_10_(—)p2_640x360.yuv</LocalObjectLocation> </VideoSource> <!-- Ad videoInitially disabled--> <VideoSource Xpos=“200” Ypos=“28” VideoID=“11”DepthOrder=“2” Width=“480” Height=“360” RenderedWidth=“878”RenderedHeight=“658” AlphaBlend=“175” StartPlaying=“0”SyncStreamIDs=“21” FrameRate=“23.976” > <LocalObjectLocation>null.yuv</LocalObjectLocation> <TargetActionTargetID=“11”/> </VideoSource> <!-- Main Audio --> <AudioSourceAudioID=“20” StartPlaying=“true”> <LocalObjectLocation>E:\VIDSRC\HDtoSD\NASCAR\rac\rac_01_09_(—)40.wav</LocalObjectLocation> </AudioSource> <!-- Ad-Audio --><AudioSource AudioID=“21” StartPlaying=“false”> <LocalObjectLocation>null</LocalObjectLocation> </AudioSource> <!--First Ad Image Initially disabled AlphaBlend --> <ImageOverlayContentType=“3” Duration=“0” Visible=“0” DepthOrder=“3” ID=“30”AlphaBlend=“255”> <BoundingBox TopLeftX=“31” TopLeftY=“28”BottomRightX=“95” BottomRightY=“92” /> <!-- <Animation Duration=“1000”FinalXPos=“64” FinalYPos=“40” LoopAnimation=“false” /> --> <TargetActionTargetID=“11”/>  <LocalObjectLocation>null.png</LocalObjectLocation></ImageOverlay> <MetaData Name=“ADWD” ImageObjects=“30”VideoObjects=“11” AudioObjects=“21” MainVideoId=“10” MainAudioId=“20”VidDuration=“15000” Duration=“500000”StartOffset=“600”><Text>beer</Text></MetaData> </Scene> <!-- McDonaldAd2 --> <Scene ID=“100”> <!-- Main video --> <VideoSource Xpos=“31”Ypos=“28” VideoID=“110” DepthOrder=“1” Width=“640” Height=“360”RenderedWidth= “1214” RenderedHeight=“658” SyncStreamIDs=“120”FrameRate=“59.97” > <LocalObjectLocation>F:\DEMO_F\NGV\Nascar\rac_02_11_27a_10_(—)p2_640x360.yuv</LocalObjectLocation> </VideoSource> <!-- Ad videoInitially disabled--> <VideoSource Xpos=“31” Ypos=“28” VideoID=“111”DepthOrder=“2” Width=“480” Height=“360” RenderedWidth=“878”RenderedHeight=“658” AlphaBlend=“175” StartPlaying=“0”SyncStreamIDs=“121” FrameRate=“23.976” > <LocalObjectLocation>null.yuv</LocalObjectLocation> <AnimationDuration=“30000” FinalXPos=“380” FinalYPos=“28” /> <TargetActionTargetID=“111”/> </VideoSource> <!-- Main Audio --> <AudioSourceAudioID=“120” StartPlaying=“true”> <LocalObjectLocation>E:\VIDSRC\HDtoSD\NASCAR\rac\rac_02_11_(—)27.wav</LocalObjectLocation> </AudioSource> <!-- Ad-Audio --><AudioSource AudioID=“121” StartPlaying=“false”> <LocalObjectLocation>null</LocalObjectLocation> </AudioSource> <!--First Ad Image Initially disabled AlphaBlend --> <ImageOverlayContentType=“3” Duration=“2000000” Visible=“0” DepthOrder=“3” ID=“130”AlphaBlend=“255”> <BoundingBox TopLeftX=“31” TopLeftY=“28”BottomRightX=“95” BottomRightY=“92” /> <TargetAction TargetID=“111”/> <LocalObjectLocation>null.png</LocalObjectLocation> </ImageOverlay><RMIRectangle FrameRectangle=“0” ID=“140” FillRectangle=“0”DepthOrder=“3” FillR=“200” FillG=“20” FillB=“20”> <BoundingBoxTopLeftX=“828” TopLeftY=“74” BottomRightX= “1027”BottomRightY=“121”></BoundingBox> <TargetAction TargetID=“111”/></RMIRectangle> <MetaData Name=“ADWD” ImageObjects=“130”VideoObjects=“111” AudioObjects=“121” MainVideoId=“110”MainAudioId=“120” VidDuration=“25000” Duration=“500000”StartOffset=“600”><Text>burger</Text></MetaData> </Scene> <ImageOverlayContentType=“3” Duration=“0” ID=“92” DepthOrder=“3” Visible=“1” ><BoundingBox TopLeftX=“616” TopLeftY=“688” BottomRightX=“640”BottomRightY=“712” /> <!-- <TargetAction TargetID=“900”/> --><TargetAction HideTargets=“0 100 200 300 400 500” ShowTargets=“900” /><LocalObjectLocation>E:\DEMO\RMI\image\32px-Crystal_Clear_app_noatun.png</LocalObjectLocation> </ImageOverlay><ImageOverlay ContentType=“3” Duration=“0” ID=“93” DepthOrder=“0”Visible=“1”> <BoundingBox TopLeftX=“0” TopLeftY=“0” BottomRightX=“1280”BottomRightY=“832”/> <LocalObjectLocation>E:\DEMO\RMI\image\TV-MU-60PZ95V_LG.jpg</LocalObjectLocation> </ImageOverlay> <TextOverlayID=“94” DepthOrder=“3” Duration=“0” Visible=“true”> <BoundingBoxTopLeftX=“30” TopLeftY=“30” BottomRightX=“450” BottomRightY=“130” /><Color R=“250” G=“250” B=“250”/> <Font FamilyName=“Lucida Console”Size=“28”/> <!-- Horizontal animation --> <Animation Duration=“40000”FinalXPos=“830” FinalYPos=“655” LoopAnimation=“true” /> <Text>NGVInteractive Ads</Text> </Text Overlay> </RMIInstance> </RMI>

The example above illustrates exemplary RMI elements that may berequired to define an RMI presentation, including headers, a<Canvas>element, a <VideoSource>element, an <AudioSource>element, and a<Text>element. The example also illustrates an example of a chain ofvideo content <VideoChain>, as well as video content (e.g., a Videoelement) with alpha-blending attributes and definition of genericgraphics polygon <RMIPolygon>. The portion of the presentation shownhere is part of full presentation that corresponds to the screenshotdepicted in FIG. 13A.

The following is a second example of RMI presentation comprising RMIinstances:

<?xml version=“1.0” encoding=“UTF-8”?><RMIxmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance”xsi:noNamespaceSchemaLocation=“file:///F:/DEMO_F/RMI/2008-09-10-RMIPlayerDX/rmi104.xsd” FrameRate=“30”>  <Canvas Width=“1280”Height=“720”> <Color R=“128” G=“128” B=“128”/>  </Canvas>  <!-- Usage ofScene element -->  <RMIInstance FrameOffset=“10”> <Scene ID=“450”><VideoSource Xpos=“1200” Ypos=“0” VideoID=“300” DepthOrder=“3”Width=“352” Height=“288” RenderedWidth=“704” RenderedHeight=“576”FrameRate=“29.97” LoopVideo=“1”><LocalObjectLocation>E:\VIDSRC\CIF\Dancer\dancer_(—)cif_1.yuv</LocalObjectLocation> <TransparentColor MinR=“55” MinG=“55”MinB=“55” MaxR=“85” MaxG=“85” MaxB=“85”/> <Animation Duration=“2000”FinalXPos=“280” FinalYPos=“100” /> </VideoSource> <VideoSourceXpos=“800” Ypos=“0” VideoID=“301” DepthOrder=“3” Width=“352”Height=“288” AlphaMask=“E:\VIDSRC\CIF\Dancer\dancer_cif_1.seg”AlphaBlend=“150” FrameRate=“29.97” StartOffset=“100” LoopVideo=“1”><LocalObjectLocation>E:\VIDSRC\CIF\Dancer\dancer_(—)cif.yuv</LocalObjectLocation> <AlphaAnimation Duration=“5000”InitialAlpha=“225” FinalAlpha=“0” LoopAnimation=“1”/> </VideoSource><!-- Image is the next slide button --> <ImageOverlay ContentType=“3”Duration=“9000” Visible=“1” ID=“105” DepthOrder=“2” AlphaBlend=“255”><BoundingBox TopLeftX=“1248” TopLeftY=“694” BottomRightX=“1268”BottomRightY=“716”/> <TransparentColor MinR=“255” MinG=“255” MinB=“255”/> <TargetAction MouseOver=“1” TargetID=“451”/><LocalObjectLocation>E:\DEMO\RMI\image\32px-Go-down-rot_svg.png</LocalObjectLocation> </ImageOverlay> </Scene><TextOverlay ID=“94” DepthOrder=“3” Duration=“0” Visible=“1”><BoundingBox TopLeftX=“30” TopLeftY=“350” BottomRightX=“680”BottomRightY=“430” /> <Color R=“20” G=“20” B=“250”/> <FontFamilyName=“Chicago” Size=“36” Style=“italic” /> <!-- Horizontalanimation -->  <Animation Duration=“50000” FinalXPos=“635”FinalYPos=“350” LoopAnimation=“true” /> <Text>NGV InteractiveObjects/ROI</Text> </TextOverlay> <!-- Rectangle --> <RMIRectangleFrameRectangle=“1” ID=“95” FillRectangle=“1” DepthOrder=“2” Duration=“0”FillR=“60” FillG=“60” FillB=“60”> <BoundingBox TopLeftX=“31”TopLeftY=“28” BottomRightX=“1245” BottomRightY=“688”></BoundingBox></RMIRectangle> </RMIInstance> </RMI>

This example illustrates the use of a <scene>RMI element that specifiesa grouping of audio, video, image and text content assets. An imagecontent asset may be specified by an <ImageOverlay>RMI element. Theaddition of a <TargetAction>attribute to an RMI element specifyingimage, video, text or graphics content allows behavior to be added tothe element, such as a change to the state of the element (e.g., turn iton/off, which may cause the corresponding element to the added and/orremoved from a display). The example also illustrates the combination ofa first content asset with an additional multimedia element item using<VideoSource>and <LocalObjectLocation>elements. The elements andattributes associated with the secondary item (dancer animation“dancer_cif_1.yuv”), may define how the element is combined (e.g., usingan <AlphaAnimation>element) with other video content, and the like. Theexample further illustrates the use of user-interaction components. A“next slide button” is defined using an <ImageOverlay>RMI elementreferencing a button image in PNG image format. The example furtherspecifies actions to be taken when a user interacts with theuser-interaction element (e.g., using the <TargetAction>RMI element).

FIGS. 9A and 9B depict screen shots of an RMI presentation in whichinteractive scenes of windowed video-wall-like playback of four (in FIG.9A) and two (in FIG. 9B) videos of the same scene are displayed. Thispresentation of FIG. 9 was constructed using RMI elements disclosedherein.

FIGS. 10A, 10B, and 10C depict screen shots of an RMI presentationcomprising RMI interactive scenes showing three out of ten videoscalability layers available of the same scene. A first layer isdepicted in FIG. 10A, a second in FIG. 10B, and a third in FIG. 10C. TheRMI presentation depicted in FIGS. 10A-10C was constructed using RMIelements taught herein.

FIG. 11 depicts screen shots of an RMI presentation in which fortychapters of a movie as shown as a plurality of “live chapter” selectableregions (hotspots) 1110 that, when selected by a user, cause full sizevideos of the corresponding chapter to be displayed. The RMIpresentation depicted in FIG. 11 was constructed using the RMI schemataught herein.

FIGS. 12A, 12B, and 12C depict screen shots of an RMI presentationcomprising interactive video object overlay elements 1210, 1220, and1230. Some of the objects 1210, 1220, and/or 1230 may be selected by auser. Selection of the objects 1210, 1220, and/or 1230 may causemodifications to the RMI presentation, such as enabling/disablingvarious elements of the RMI presentation, playback of additionalelements, element animation and/or movement, replacement of a backgroundregion, or the like. Video objects, such as the objects 1210, 1220,and/or 1230 may be pasted into new scenes and blended to appear asnormal part of this scene yet still remain selectable. Clicking withinon a background region of a scene (a hotspot or ROI within the scenebackground) may cause a video object that was not previously visible tobe displayed.

FIGS. 13A, 13B, and 13C depict screen shots of RMI presentation scenescomprising advertising multimedia elements 1310, 1320, and 1330 insertedtherein. The advertisement multimedia elements 1310, 1320, and/or 1330may be interactive. For example, an interactivity icon (not shown) maybe provided on one or more of the display areas. Selection of the iconmay trigger the display of an advertising multimedia element 1310, 1320,and/or 1330. Alternatively, the display of an advertising multimediaelement 1310, 1320, and/or 1330 may be triggered automatically (e.g.,according to embedded ROI content-aware metadata or the like). In someembodiments, the advertisement multimedia elements 1310, 1320, and/or1330 may be dismissed by selecting an area within the advertising imageoverlay (e.g., a hotspot on the image). As shown in FIG. 13, theadvertisement multimedia elements 1310, 1320, and/or 1330 may besemi-transparent, to allow other programming to be viewed concurrentlywith the advertising (e.g., viewed concurrently with a first contentasset).

Various aspects of the disclosure may include a multimedia/videobrowsing system (as compared to HTML, a text based browsing system withmultimedia additions) that allows for browsing content on an objectand/or ROI basis (e.g., via the hotspots embedded within the content).An interactive video/multimedia presentation (as defined by IVMA data)may be presented using a standalone client application (e.g., on auser's laptop, PC, mobile phone or portable device (PMP)) and/or may bepresented in conjunction with a server, such as a web server (e.g., runremotely within a portal-type application).

Another aspect of the disclosure may include an Advanced MultimediaAuthoring Language for defining an interactive video/multimediapresentation. The language may work with compressed or uncompressedcontent and may provide mechanisms for advanced rendering, compositionand interactivity. The RMI schema disclosed herein is one example ofsuch a language.

In one embodiment, IVMA data may be created with a text editor or otherauthoring tool comprising a GUI, IDE, or the like. The IVMA data may beinterpreted by a conforming playback device or application thatimplements the IVMA schema (e.g., interprets the IVMA data, requestscontent assets, renders and composes assets, and provides for userinteraction according to the IVMA data).

IVMA data may be transmitted to a client as a standalone file or stream.Alternatively, IVMA data may be segmented and embedded in raw videoframes, or in a compressed video stream of a video (or audio) codec.

A binary compressed representation of IVMA data may allow for efficientstorage or transmission of the presentation. The presentation, whether,for example, in human readable format, such as XML or in machinereadable binary format, may be transmitted in its entirety at one timeor distributed throughout content, such as video or audio.

In one embodiment, an interactive video/multimedia presentation (asdefined by IVMA data) combines interactivity, scalability andobject/region of interest (ROI) based features. Integration ofJava/ECMAScript may be provided in the IVMA data to provide additionalflexibility for animation as well as dynamic changes initiated by a useror otherwise. For example the Java/ECMAScript may allow a user tointeract with 3-D models (or other multimedia elements) within the IVMapplication.

The above description provides numerous specific details for a thoroughunderstanding of the embodiments described herein. However, those ofskill in the art will recognize that one or more of the specific detailsmay be omitted, or other methods, components, or materials may be used.In some cases, operations are not shown or described in detail.

Furthermore, the described features, operations, or characteristics maybe combined in any suitable manner in one or more embodiments. It willalso be readily understood that the order of the steps or actions of themethods described in connection with the embodiments disclosed may bechanged as would be apparent to those skilled in the art. Thus, anyorder in the drawings or Detailed Description is for illustrativepurposes only and is not meant to imply a required order, unlessspecified to require an order.

Embodiments may include various steps, which may be embodied inmachine-executable instructions to be executed by a general-purpose orspecial-purpose computer (or other electronic device). Alternatively,the steps may be performed by hardware components that include specificlogic for performing the steps or by a combination of hardware,software, and/or firmware.

Embodiments may also be provided as a computer program product includinga computer-readable medium having stored thereon instructions that maybe used to program a computer (or other electronic device) to performprocesses described herein. The computer-readable medium may include,but is not limited to: hard drives, floppy diskettes, optical disks,CD-ROMs, DVD-ROMs, ROMs, RAMs, EPROMs, EEPROMs, magnetic or opticalcards, solid-state memory devices, or other types ofmedia/machine-readable medium suitable for storing electronicinstructions.

As used herein, a software module or component may include any type ofcomputer instruction or computer executable code located within a memorydevice and/or transmitted as electronic signals over a system bus orwired or wireless network. A software module may, for instance, compriseone or more physical or logical blocks of computer instructions, whichmay be organized as a routine, program, object, component, datastructure, etc. that performs one or more tasks or implements particularabstract data types.

In certain embodiments, a particular software module may comprisedisparate instructions stored in different locations of a memory device,which together implement the described functionality of the module.Indeed, a module may comprise a single instruction or many instructions,and may be distributed over several different code segments, amongdifferent programs, and across several memory devices. Some embodimentsmay be practiced in a distributed computing environment where tasks areperformed by a remote processing device linked through a communicationsnetwork. In a distributed computing environment, software modules may belocated in local and/or remote memory storage devices. In addition, databeing tied or rendered together in a datastore record may be resident inthe same memory device, or across several memory devices, and may belinked together in fields of a record in a datastore across a network.

It will be understood by those having skill in the art that many changesmay be made to the details of the above-described embodiments withoutdeparting from the underlying principles of the invention. The scope ofthe present invention should, therefore, be determined only by thefollowing claims.

What is claimed is:
 1. An apparatus comprising: processing logic toautomatically detect an object and its location in a plurality of framesof video while the video is playing, and to associate content with thedetected object using metadata, wherein the metadata associates thelocation of the object in the frames to content to provide formetadata-to-content synchronization for said frames, and upon renderingof the frames, said processing logic to add the content to the frames,based on the metadata, at the locations in the frames, in associationwith the detected object; and memory coupled to the processing logic,the memory to store the frames.
 2. The apparatus of claim 1, saidprocessing logic to automatically detect an object by detecting at leastone of a shape or texture.
 3. The apparatus of claim 1, wherein themetadata comprises at least one of a region of interest (ROI) boundingthe detected object, and a location of the detected object in the frame.4. The apparatus of claim 1, said processing logic to track movement ofthe object and to display the content on the object as it moves.
 5. Theapparatus of claim 1, wherein said content is an image.
 6. The apparatusof claim 5, wherein said image is an advertisement.
 7. The apparatus ofclaim 1, wherein said content is an advertisement.
 8. The apparatus ofclaim 1, wherein said content is text.
 9. A method comprising:automatically detecting an object and its location in a plurality offrames of video while the video is playing; associating content with thedetected object using metadata, wherein the metadata associates thelocation of the object in the frames to content to provide formetadata-to-content synchronization for said frames, and upon renderingof the frames; and adding the content to the frames, based on themetadata, at the locations in the frames, in association with thedetected object.
 10. The method of claim 9, including detecting anobject by detecting at least one of a shape or texture.
 11. The methodof claim 9, wherein the metadata comprises at least one of a region ofinterest (ROI) bounding the detected object, and a location of thedetected object in the frame.
 12. The method of claim 9, includingtracking movement of the object and to display the content on the objectas it moves.
 13. The method of claim 9, wherein said content is animage.
 14. The method of claim 13, wherein said image is anadvertisement.
 15. The method of claim 9, wherein said content is anadvertisement.
 16. The method of claim 9, wherein said content is text.17. One or more non-transitory computer readable media storinginstructions to perform a sequence comprising: automatically detectingan object and its location in a plurality of frames of video while thevideo is playing; associating content with the detected object usingmetadata, wherein the metadata associates the location of the object inthe frames to content to provide for metadata-to-content synchronizationfor said frames, and upon rendering of the frames, and adding thecontent to the frames, based on the metadata, at the locations in theframes, in association with the detected object.
 18. The media of claim17, further storing instructions to automatically detect an object bydetecting at least one of a shape or texture.
 19. The media of claim 17,wherein the metadata comprises at least one of a region of interest(ROI) bounding the detected object, and a location of the detectedobject in the frame.
 20. The media of claim 17, further storinginstructions to track movement of the object and to display the contenton the object as it moves.
 21. The media of claim 17, wherein saidcontent is an image.
 22. The media of claim 21, wherein said image is anadvertisement.
 23. The media of claim 17, wherein said content is anadvertisement.
 24. The media of claim 17, wherein said content is text.25. An apparatus comprising: means for automatically detecting an objectand its location in a plurality of frames of video while the video isplaying; means for associating content with the detected object usingmetadata, wherein the metadata associates the location of the object inthe frames to content to provide for metadata-to-content synchronizationfor said frames, and upon rendering of the frames; and means for addingthe content to the frames, based on the metadata, at the locations inthe frames, in association with the detected object.
 26. The apparatusof claim 25, including means for detecting an object by detecting atleast one of a shape or texture.
 27. The apparatus of claim 25, whereinthe metadata comprises at least one of a region of interest (ROI)bounding the detected object, and a location of the detected object inthe frame.
 28. The apparatus of claim 25, including means for trackingmovement of the object and to display the content on the object as itmoves.
 29. The apparatus of claim 25, wherein said content is an image.30. The apparatus of claim 25 wherein said image is an advertisement.